CLAIMS 

Having thus described our invention, what we claim as new and desire to secure by Letters Patent 
is as follows: 

1. A system for automatically detecting neutral, expressionless face images in digital images, the 
system comprising: 

an image acquisition unit; 

an face detector receiving input from the image acquisition unit, detecting one or more face 
subimages of one or more faces in the image; 

a characteristic point detector that receives input from the face detector and that estimates one or 
more characteristic facial features as characteristic points in each detected face subimage, one of 
the characteristic points being a first and a second mouth corner on a mouth of the face; 

a facial feature detector that detects one or more contours of one or more facial components; 

a facial feature analyzer that determines a mouth shape of the mouth from the contour of the 
mouth and creates a representation of the mouth shape, the mouth being one of the facial 
components; and 
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a face classification unit that classifies the representation into one of a neutral class and a 
non-neutral class. 

2. A system, as in claim 1, where the mouth shape is determined by a width of the mouth, the 
first distance from the first mouth corner to a reference in the coordinate systems, and the second 
distance from the second mouth corner to the reference in the coordinate systems. 

3. A system, as in claim 1, where one or more of the characteristic facial features is an eye 
openness measure of an eye of the facial image and the representation includes the eye openness 
measure. 

4. A system, as in claim 1, where one or more characteristic facial features is derived from any 
one of an eye region and an eyebrow region in the face subimage. 

5. A system, as in claim 1, where the contours are extracted by an edge detector. 

6. A system, as in claim 1, where the characteristic facial features are estimated by template 
matching techniques. 

7. A system, as in claim 1, where the facial components include any one or more of the 
following: a mouth, an eye, an eyebrow, a nose, an ear, and a wrinkle. 
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8. A system, as in claim 1, where a mouth position of the mouth is determined with respect to a 
geometric reference. 

9. A system, as in claim 1, where a mouth position of the mouth is determined with respect to an 
image coordinate system. 

10. A system, as in claim 1, where the characteristic points define one or more characteristic 
distances to one or more respective geometric references. 

1 1. A system, as in claim 10, where one or more of the characteristic distances is normalized to 
another of the characteristic distances. 

12. A system, as in claim 1, where the characteristic points further include a location of each eye 
on the face, being eye characteristic points and one of the characteristic distances is a distance 
between the eye characteristic points, being an eye separation distance. 

13. A system, as in claim 12, where one or more of the other characteristic distances is 
normalized by the eye separation distance. 

14. A system, as in claim 12, where the eye characteristic points are located on the eye pupil. 



15. A system, as in claim 10, where the facial feature analyzer divides the face sub-image into 
two or more zones. 
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16. A system, as in claim 15, where facial feature analyzer determines one or more component 
shapes from one or more respective component contours in one or more of the zones. 

17. A system, as in claim 16, where the component is the mouth and the shape of the mouth is 
extracted from three of the zones containing the mouth. 

18. A system, as in claim 17, where the shape of the mouth is determined by histograming the 
contours of the mouth in each of the three zones containing the mouth. 

19. A system as in claim 18 where the mouth shape is defined by three shape histograms each 
corresponding to one of the three zones containing the mouth obtained by quantizing the contour 
of the mouth into four directions. 

20. A system, as in claim 17, where a further component is one or more of an eye and an eyebrow 
that are extracted from three of the zones containing the eyes. 

2L A system, as in claim 10, where one or more of the characteristic distances is the normalized 
width of the mouth, the width of the mouth being the distance between the first and second 
mouth corners. 



YOR9-2002-0007US1 



40 



22. A system, as in claim 3, where the eye openness measure of the eyes is measured by a 
normalized height of the eyes, 

23. A system, as in claim 12, where a mouth-eye distance between the first mouth corner and a 
first eye is normalized by the eye separation distance, and the normalized mouth-eye distance is 
an additional feature used by the face classification unit. 

24. A system, as in claim 12, where a brow-eye distance between a first eyebrow and a first eye is 
normalized by the eye separation distance, and the normalized brow-eye distance is an additional 
feature used by the face classification unit. 

25. A system, as in claim 12, where a brow distance between the first eyebrow and the second 
eyebrow is normalized by the eye separation distance, and the normalized brow distance is an 
additional feature used by the face classification unit. 

26. A system, as in claim 1, where one or more normalized characteristic distances is used by the 
face classification unit classify if the face is neutral. 

27. A system, as in claim 26, where the face classification unit is one or more of the following 
classifiers: neural networks, nearest neighbor methods, Bayesian inference methods, linear 
discriminant methods, perceptrons, support vector machines, and related variants. 



28. A system, as in claim 26, where the face classification unit is neural network (NN). 
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29. A system, as in claim 1, where the face classification unit is used to enroll one or more 
subjects in a face-based authentication system with a neutral face. 

30. A system, as in claim 1, where the face classification unit is used to authenticate one or more 
subjects in a face-based authentication system with a neutral face. 

31. A system, as in claim 1, used as a user interface that reacts to one or more human emotion 
states. 

32. A system, as in claim 1, used as a surveillance device. 

33. A system, as in claim 32, used in one or more of the following: a financial institution, a bank 
a retail store, a credit card check, a check in, an air port, an air plane boarding check, a building 
access system, and a object access authorization system. 

34. A system, as in claim 1, used as an information retrieval device. 

35. A system, as in claim 34, where the information retrieval device includes any one or more of 
the following: a video retrieval of a frame containing a face image with a specified expression, an 
image of an image containing a face image with a specified expression. 
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36. A method for automatically detecting neutral, expressionless face images in digital images, 
the method comprising the steps of: 

acquiring an image; 

detecting one or more face subimages of one or more faces in the image; 

estimating one or more characteristic facial features as characteristic points in each detected face 
subimage, one of the characteristic points being a first and a second mouth corner on a mouth of 
the face; 

detecting one or more contours of one or more facial components; 
determining a mouth shape of the mouth from the contour of the mouth; 

creating a representation of the mouth shape, the mouth being one of the facial components; and 
classifying the representation into one of a neutral class and a non-neutral class. 

37. A system for automatically detecting neutral, expressionless face images in digital images, 
the method comprising: 



means for acquiring an image; 
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means for detecting one or more face subimages of one or more faces in the image; 

means for estimating one or more characteristic facial features as characteristic points in each 
detected face subimage, one of the characteristic points being a first and a second mouth corner 
on a mouth of the face; 

means for detecting one or more contours of one or more facial components; 

means for determining a mouth shape of the mouth from the contour of the mouth; 

means for creating a representation of the mouth shape, the mouth being one of the facial 
components; and 

means for classifying the representation into one of a neutral class and a non-neutral class. 

38. A computer program product having a computer program for automatically detecting neutral, 
expressionless face images in digital images, the computer program comprising the steps of: 

acquiring an image; 

detecting one or more face subimages of one or more faces in the image; 
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estimating one or more characteristic facial features as characteristic points in each detected face 
subimage, one of the characteristic points being a first and a second mouth corner on a mouth of 
the face; 

detecting one or more contours of one or more facial components; 

determining a mouth shape of the mouth from the contour of the mouth; 

creating a representation of the mouth shape, the mouth being one of the facial components; and 

classifying the representation into one of a neutral class and a non-neutral class. 
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